Adaptive private network asyncrhonous distributed shared memory services

ABSTRACT

A highly predicable quality shared distributed memory process is achieved using less than predicable public and private internet protocol networks as the means for communications within the processing interconnect. An adaptive private network (APN) service provides the ability for the distributed memory process to communicate data via an APN conduit service, to use high throughput paths by bandwidth allocation to higher quality paths avoiding lower quality paths, to deliver reliability via fast retransmissions on single packet loss detection, to deliver reliability and timely communication through redundancy transmissions via duplicate transmissions on high a best path and on a most independent path from the best path, to lower latency via high resolution clock synchronized path monitoring and high latency path avoidance, to monitor packet loss and provide loss prone path avoidance, and to avoid congestion by use of high resolution clock synchronized enabled congestion monitoring and avoidance.

PRIORITY CLAIM

This application is a divisional of U.S. patent application Ser. No.16/383,262, filed on Apr. 12, 2019, which is a divisional of U.S. patentapplication Ser. No. 15/180,478, filed on Jun. 13, 2006, which is adivisional of U.S. patent application Ser. No. 14/291,776 filed on May30, 2014 (now U.S. Pat. No. 9,392,061), which is a continuation of U.S.patent application Ser. No. 13/850,411 filed on Mar. 26, 2013 (now U.S.Pat. No. 8,775,547), which is a continuation of U.S. patent applicationSer. No. 13/208,825 filed on Aug. 12, 2011 (now U.S. Pat. No.8,452,846), which claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/372,904 entitled “Adaptive Private NetworkAsynchronous Distributed Shared Memory Services” filed Aug. 12, 2010,the disclosures of all of which are hereby incorporated by reference intheir entireties.

CROSS REFERENCE TO RELATED APPLICATIONS

U.S. patent application Ser. No. 13/592,460 filed on Aug. 23, 2012 (nowU.S. Pat. No. 8,644,144), which is a continuation of U.S. patentapplication Ser. No. 13/353,693 filed on Jan. 19, 2012 (now U.S. Pat.No. 8,274,891), which is a continuation of U.S. patent application Ser.No. 12/482,766 filed Jun. 11, 2009 (now U.S. Pat. No. 8,125,907), all ofwhich claim the benefit of U.S. Provisional Patent Application Ser. No.61/060,846 entitled “Flow-based Adaptive Private Network with MultipleWAN-Paths” filed Jun. 12, 2008 have the same assignee as the presentapplication, are related applications, and are hereby incorporated byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to improved networkcommunication in loosely coupled computing systems. More specifically,the present invention relates to systems and methods for effectivelyemploying distributed shared memory services for adaptive privatenetworks.

BACKGROUND OF THE INVENTION

The introduction of frame relay in the early 1990's brought lower cost,higher bandwidth, improved reliability, and simpler management controlto enterprise wide area networks (WANs) as compared to X.25 andpoint-to-point leased-line alternatives. Frame relay, together withsingle-source asynchronous transfer mode (ATM) and multiprotocol labelswitching (MPLS) services, still dominate the enterprise WAN market forcorporate Internet traffic. However, these Internet-based services arenot as reliable as processing systems at network nodes and are difficultto predict performance of network traffic. In addition, many networksexperience disruptions and congestion on individual paths, for example,due to old equipment and marginal cable and connector conditions. Havingsuch dependencies on single network paths, causes network outages andlow performance even with new installations.

While performance, reliability, and predictability of a network hasimproved due to improvements in processor and communicationarchitectures and implementations, the underlying networks and messagepassing protocols remain inefficient and hold back potential performanceimprovements.

SUMMARY OF THE INVENTION

Among its several aspects, the present invention addresses systems andtechniques which improve performance, reliability, and predictability ofnetworks without having costly hardware upgrades or replacement ofexisting network equipment. To such ends, an embodiment of the inventionaddresses a method for using distributed shared memory for communicatingbetween loosely coupled computing systems. A first time is calibrated ina first node and a second time is calibrated in a second node based on acurrent time received from a network control point (NCP). A block ofmemory is transmitted from the first node over a first communicationpath to the second node and a duplicate block of memory is transmittedfrom the first node over a second communication path having a measure ofindependence from the first communication path to the second node thatis greater than measured levels of independence of other availablecommunication paths, wherein the block of memory and the duplicate blockof memory are both tagged with a corresponding send time according tothe calibrated first time. The block of memory or the duplicate block ofmemory is selected as the first block of memory received in the secondnode and tagged with an arrival time according to the calibrated secondtime, wherein the send time and arrival time of the first block ofmemory received in the second node are analyzed taking into accountprevious transmission measurements at the second node and used togenerate a second node path quality report. The second node path qualityreport is transmitted from the second node to the first node, whereinpath selection for transmission of a second block of memory is based onthe second node path quality report.

Another embodiment addresses a method of sharing memory in a multipleparallel path network having disparate communication paths between atransmitting network node and a receiving network node. In atransmitting node a report on multiple disparate communication pathsassociated with the transmitting node is compressed to a compressedreport. The compressed report having a transmission size exceeding atransmission capacity of a first communication path is fragmented into afirst fragment and a second fragment. The first fragment is transmittedacross the first communication path to a receiving node and the secondfragment is transmitted across a second communication path having ameasure of independence from the first communication path to thereceiving node that is greater than measured levels of independence ofother available communication paths. The first fragment and the secondfragment are received in the receiving node as a combined compressedreport. The combined compressed report is decompressed in the receivingnode to the report, wherein report data from the report is used toadjust selection of communication paths.

Another embodiment addresses a method for adaptively sharing memory in anetwork utilizing characterizations of multiple disparate communicationpaths across the network which varies in transmission behaviorfrequently over time. Fragment packets of a block of memory aretransmitted from a first node over a first path to a second node. Thefragment packets of the block of memory are received in the second node.A potential lost packet incident is detected in the second node. Apotential lost packet indication is transmitted in response to thepotential lost packet incident from the second node over a second pathto the first node and a duplicate potential lost packet indication istransmitted from the second node over a third path to the first node.The potential lost packet is speculatively retransmitted from the firstnode over a fourth path to the second node in response to the potentiallost packet indication or the duplicate potential lost packet indicationreceived first in the first node. The received fragment packetsincluding the speculatively retransmitted potential lost packet arecombined to recreate the block of memory.

A more complete understanding of the present invention, as well as otherfeatures and advantages of the invention, will be apparent from thefollowing detailed description, the accompanying drawings, and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention will become more fully apparentfrom the following description and appended claims, taken in conjunctionwith the accompanying drawings. Understanding that these drawings depictonly exemplary embodiments and are, therefore, not to be consideredlimiting of the invention's scope, the exemplary embodiments of theinvention will be described with additional specificity and detailthrough use of the accompanying drawings in which:

FIG. 1 illustrates a processing complex using synchronous shared memoryfor internal control of network adapters and various peripherals;

FIG. 2A illustrates a node processor complex suitable for use as asender node or as a receiver node in an adaptive private network (APN)in accordance with the present invention;

FIG. 2B illustrates a network configuration having a sender node and aplurality of receiver nodes in accordance with the present invention;

FIG. 3 illustrates software communication layers in a sender node and areceiver node using shared distributed memory in accordance with thepresent invention;

FIGS. 4A-4D illustrate a shared memory block sending process inoperation at a sending node in accordance with the present invention;

FIGS. 5A and 5B illustrate a shared memory block receiving process inoperation at a receiver node in accordance with the present invention;and

FIG. 6 illustrates a high availability (HA) network configuration havingan active node, a standby node, and a plurality of receiver nodes usingsend/receive transport reliable protocol (TRP) blocks in accordance withthe present invention.

DETAILED DESCRIPTION

The present invention will now be described more fully with reference tothe accompanying drawings, in which several embodiments of the inventionare shown. This invention may, however, be embodied in various forms andshould not be construed as being limited to the embodiments set forthherein. Rather, these embodiments are provided so that this disclosurewill be thorough and complete, and will fully convey the scope of theinvention to those skilled in the art.

It will be appreciated that the present disclosure may be embodied asmethods, systems, or computer program products. Accordingly, the presentinventive concepts disclosed herein may take the form of a hardwareembodiment, a software embodiment or an embodiment combining softwareand hardware aspects. Furthermore, the present inventive conceptsdisclosed herein may take the form of a computer program product on acomputer-readable storage medium having non-transitory computer-usableprogram code embodied in the medium. Any suitable computer readablemedium may be utilized including hard disks, CD-ROMs, optical storagedevices, flash memories, or magnetic storage devices.

Computer program code or software programs that are operated upon or forcarrying out operations according to the teachings of the invention maybe written in a high level programming language such as C, C++, JAVA®,Smalltalk, JavaScript®, Visual Basic®, TSQL, Perl, use of .NET™Framework, Visual Studio® or in various other programming languages. Aprogram written in one of these languages is compiled to a targetprocessor architecture operating on a node by converting the high levelprogram code into a native assembler program. Programs for the targetprocessor architecture may also be written directly in a nativeassembler language. A native assembler program uses instruction mnemonicrepresentations of machine level binary instructions. Program code orcomputer readable medium as used herein refers to code whose format isunderstandable by a processor. Software embodiments of the disclosure donot depend upon their implementation with a particular programminglanguage.

The methods described in connection with the embodiments disclosedherein may be embodied directly in hardware, in a software module, or ina combination of the two. A software module storing non-transitorysignals executed by a processor may reside in RAM memory, flash memory,ROM memory, EPROM memory, EEPROM memory, registers, hard disk, aremovable disk, a CD-ROM, or any other form of storage medium known inthe art. A computer-readable storage medium may be coupled to theprocessor through local connections such that the processor can readinformation from, and write information to, the storage medium orthrough network connections such that the processor can downloadinformation from or upload information to the storage medium. Thestorage medium coupling to the processor may be a direct couplingintegral to a circuit implementation or may utilize one or moreinterfaces, supporting direct accesses or data streaming usingdownloading techniques. In the alternative, the storage medium may beintegral to the processor.

The present invention is directed towards providing a flow-based,reliable, high-bandwidth network comprised of multiple paths betweensites. A node or node processor complex is an independent processingentity, which is for example, a sub-system of a network embodiment thatincludes a CPU, a memory sub-system, an operating system, and apersistent storage media such as a hard disk. The network embodimentincludes a high performance interconnect structure having means throughwhich the nodes or independent processing entities communicate state andsynchronization information for the purpose of executing distributedapplications. A distributed application is an application that operatesacross multiple independent processor entities. The distributedapplication may be highly responsive or may be batch oriented withvariations in between. For example, a highly responsive distributedapplication is generally required to respond to requests for serviceswithin a sub second response time with a very high probability ofmeeting the requirements. In contrast, a batch distributed applicationis generally required to respond to requests for services that are nottime-limited which allows responses to be non-deterministic withresponses that may vary from seconds to, for example, 30 hours orlonger.

The time responsiveness of distributed applications is limited by theability of the interconnect to provide communications between theindependent processor entities the application is distributed across.For example, if the requirement for the distributed application is to behighly responsive, the underlying interconnect is required todeterministically be of very low loss, low latency, high bandwidth, lowcongestion, and low jitter. Stated differently, for the services of thedistributed application to be timely, the underlying interconnect musthave a narrow range between best case and typical case that tends toconverge to the best case for delivery of state information betweenprocessing entities and the applications contained within them.Generally, if the required service level is not provided by theunderlying interconnect, the distributed application will not be able toprovide its users the required level of responsiveness for proper andcorrect results of the application.

There are many different types of interconnects that may be used fordistributed applications. One example is the use of a high speed buswithin a multi-core central processing system. Another example is theuse of the public Internet with applications distributed across diversegeographies and processors types.

For a highly responsive distributed application, the typicalaccommodation has been to collect all independent processor entities atthe same geographic location and interconnect the independent processorentities with high performance and high capacity connections. Examplesare processor farms and super computer clusters that use very high speednetworks or backplane buses and switches to accommodate the need formulti process memory sharing and synchronization. A famous example isthe computer system Deep Blue that combined 30 independent processingentities with a high performance interconnection containing a high speedand low latency crossbar fabric for the purpose of playing a game ofchess using a distributed application. The interconnect used, because itpossessed a highly deterministic quality, provided the ability for thedistributed application to make complex decisions in short periods oftime as was required for its application in chess matches.

An interconnect or network is generally not able to be of highdeterministic quality and distributed applications operating within astandard type network are generally not able to be highly responsive. Anetwork typically not used as an interconnect for highly responsivedistributed applications is the public Internet, because it lacks theperquisite properties of deterministic high bandwidth and high quality.If the Deep Blue distributed application attempted to utilize the publicInternet as the interconnect between its independent processing nodes,the application would not have been able to provide the timelinessrequired for its analysis routines and would not have been as successfulas a result. Typically, the public Internet may be used as aninterconnect for distributed applications as long as the distributedapplications are not required to be highly responsive. Also, a singlepublic Internet connection does not provide for high quality and highlyresponsive networks for extended periods. A distributed application thatrequires fast response times may become unreliable and prone toinconsistent timing behaviors during the duration of its run. Thepredictability of an interconnect, such as the Internet, generallydecreases more when the geographies are diverse. Even though the publicInternet generally provides the ability to connect any computer to anyother computer, by design, it does not discriminate if the computer isin Bangalore, India or New York, N.Y., USA. The performance of thepublic Internet, may also be highly inconsistent between thesegeographies and at different times of day depending on the general useof the public Internet and congestion at particular times within ageographic location and within a service provider network.

There are distributed applications that do not require highresponsiveness that currently use the public Internet to provideprocessing interconnects. These distributed applications are typicallybatch oriented and have very limited need for sub-secondsynchronization. These applications typically submit a “job” to anindependent processor across the network, and then some time later,typically minutes or hours, the job completes and replies with theresults. A famous example of batch type distributed application is theSETI@home screen saver application that uses thousands of volunteer'scomputers during their idle times to process batch jobs to analyze radiosignals from outer space. Each local job takes about 30 hours of time tocomplete.

An evaluation of the spectrum of distributed applications underlying aninterconnection's ability to provide deterministic services from highlyresponsive to the batch oriented services shows that a strongcorrelation can be found between an application's responsiveness and itsunderlying processing interconnection's ability to manage latency,bandwidth, congestion, jitter and loss. Thus, distributed applicationshave generally been designed and constrained by the capabilities ofunderlying processing interconnects, including the ability of each ofthe nodes to satisfy each application's requirements. As a consequence,if a highly predictable interconnect is not available, a distributedapplication will not be highly responsive.

As illustrated herein, a means of providing highly predictable qualityinterconnections for use with highly responsive distributed applicationsis described using the less than predicable networks, such as theInternet, private wide area networks, or networks within an enterprise.The method does this by advantageously adapting the capabilities ofadaptive private networking technology. U.S. patent application Ser. No.12/482,766 entitled “Flow-Based Adaptive Private Network With MultipleWAN-Paths”, filed on Jun. 11, 2009 which claims the benefit of61/060,846 filed Jun. 12, 2008 are both hereby incorporated by referencein their entirety. As adopted herein, this technology permits multipleindependent networks of potentially differing qualities to be combinedto provide a typical service level that is better than any one of thecomponent networks could provide if it was used solely.

The technology and methods described herein may be applied to anynetwork interconnect that may have times of relative unpredictability.Examples of non-Internet and non-wide area network applications includeutilization of various wireless networks interconnected at an enterprisepremise, infrared communications, local area networks, and networks foruse in military applications, such as on a naval ship whereinterconnects between systems generally have high amounts ofredundancies and resiliencies in case damage to differing parts of theship is sustained.

The adaptive private network (APN) contains two or more nodes that arelocated at points of entry to wide area networks (WANs) and at points ofexits from wide area networks. We refer to the point of entry to thewide area network as WAN ingress. We refer to the point of exit from thewide area networks as WAN egress. Access to wide area networks isprovided by a network service provider. The APN nodes may have multiplewide area network access points. We refer to WAN access points generallyas WAN links. It is preferred and optimal if each of the wide areaaccess points is provided by different service providers so as toprovide for higher redundancy and eliminate single points of failure.Wan links may access the public internet, such us the use of broadbandcable lines, digital subscriber lines or other. Other WAN links mayprovide private wide area networks such as multi-protocol labelswitching networks (MPLS), point to point leased circuit networks orothers. The APN establishes multiple paths between the WAN ingress WANlinks and WAN egress WAN links across the network. The term pathgenerally is used to refer to a unidirectional connection between a WANingress WAN link and a WAN egress WAN link. The term conduit is used torefer to the connection between an APN node at one location and an APNnode at another location which are connected via multiple paths.

Each node in an adaptive private network is configured to operate with ahighly calibrated shared reference clock. Within the APN, there existone or more APN nodes that serve as the reference clock source. Thesenodes are referred to as network control points (NCPs). Within thenetwork there are one or more APN nodes that serve as clients nodes andacquire their reference clock from NCP nodes. These nodes are referredto as client nodes. The APN network clock calibration is performed by aclock request message that contains the local clock time at the clientnode and a serial number that is sent from an APN client node to the NCPnode via duplicate transmission across paths that are as independent aspossible from each other. The NCP nodes reply to the first clock requestmessage received. The NCP nodes discard any other clock request messagesreceived that contains any serial number from a previously processedrequest from the APN client node. This provides for the processing ofthe clock request messages only on the lowest actual latency path. TheNCP's clock request reply message contains within it a current timeaccording to the clock of the NCP as well as a serial number. The replymessage is sent from the NCP node to the APN client node via duplicatetransmission across paths that are as independent as possible from eachother if any. The APN client accepts for processing the first clockrequest reply message received for a particular serial number. Thisprovides for the processing of the clock request reply messages on thelowest actual latency path. By reducing the round trip latency for theclock requests and replies, the APN maintains a calibrated clock withlower phase shift error than would be possible if only one path betweenclient and NCP were used.

The reference clock is used to calculate the duration of time eachpacket takes as it crosses a wide area network. The duration of packetsare used to determine statistical behavior models for latency and jitterof the network which are shared between the WAN egress nodes and the WANingress nodes via conduit quality reports. The use of the conduitquality reports are discussed below.

APN memory services utilize the underlying APN services to a highdegree. It is the use of these services that supports the APN memoryservices to be unique in its capabilities for being able to provideservices for highly responsive distributed applications across differentnetworking techniques.

With the APN memory services use of the APN services, an APN memoryrefresh may be transmitted using duplicate transmissions. When the APNtransmits data across multiple paths, the paths are chosen to be of lowlatency, low loss, and low jitter and to be from different WAN serviceproviders at the WAN ingress point and the WAN egress point if possible.If this is not possible the APN services attempts to determine uniqueservice providers at the WAN egress access points. If this is notpossible, the APN services will attempt to determine unique serviceprovides at the WAN ingress access points. By transmitting the APNmemory refresh across different high quality paths, in terms of lowloss, low latency, and low jitter, that are provided by differentservice providers when available, the APN memory refresh is able to bedelivered at lower typical latency, loss, jitter and higher availabilitythan would be possible if only one path was used. The cost of utilizingthese services is consumption of additional bandwidth for the redundanttransmissions.

With the APN memory services use of the APN services, an APN memoryrefresh may be transmitted using the APN reliable services. When the APNtransmits data across a path, the WAN egress nodes checks for packetloss via per packet sequence order accounting. The WAN egress node alsochecks for dead or quite paths with its nag protocol, which may be anindication for potential loss. Further details of loss detection and nagprotocol may be found in U.S. patent application Ser. No. 12/482,766filed Jun. 11, 2009 entitled “Flow-based Adaptive Private Network withMultiple WAN-Paths”, for example in FIG. 1H and at page 38, line 15 topage 42, line 13. If the WAN egress node detects that a packet lossoccurred or a packet loss may have occurred, the WAN egress nodeimmediately communicates the loss indication to the WAN ingress node viaa packet that is sent via duplicate transmission across paths that areas independent as possible from each other and transmitted using the APNreliable service in the opposite direction. The result is the lossindication itself is sent with high redundancy for reduced probablelatency, jitter and loss. If the packet containing the loss indicationitself is lost, the loss indication packet is retransmitted as notedbelow. The WAN ingress node, upon reception of the loss indication,retransmits the data on the current best available path for low latency,low loss, and low jitter. This retransmission may not, and typically isnot, the same path the packet was originally transmitted across when theoriginal loss occurred. The consequence of the retransmission may bethat the APN memory refresh is jittered as compared to the APN refreshif no loss had occurred. The benefit of utilization of multipleavailable paths for the retransmission is that the APN memory refresharrives with much greater reliability and lower probably of jitter thanif the packet had not been retransmitted or if the packet had been onlyretransmitted using the same path where the original loss occurred.

For example, with two APN nodes and two paths contained within theconduit. One path, Path A, has a sustained 1% packet loss and 100millisecond latency. The other path, path B, has no packet loss and 100millisecond latency. Assuming that a loss indication from the WAN egressAPN node takes 100 milliseconds to be transmitted, if the path A is usedto transmit a APN memory refresh, one APN memory refresh out of ahundred would be lost. For packets that have a successful retransmissionon the first attempt, then one packet in one hundred would have 100millisecond additional time for the retransmission plus the 100milliseconds of time for the loss indication resulting in a total jitterof 200 ms. If the retransmit of the memory refresh is constrained to usePath A, then the retransmit packet has one in a hundred chance of alsobeing lost. If the second attempt was successful, then the total jittermay be 400 milliseconds of jitter. For a network of 1% loss, thishappens once for every 10,000 packets which is considered to befrequent. Additionally, if the third retransmission is lost, then thejitter could be 600 milliseconds or more. Because the APN retransmissionthat utilize alternative paths, such as the pristine Path B above, thejitter has a substantially higher probability to be contained closer tothe ideal 200 ms than would be possible if only one path was available.

With the APN memory services use of the APN services, an APN memoryrefresh may be transmitted using the APN reliable services and usingduplicate transmissions. The cost of this very high quality service isuse of additional bandwidth and the potential for, under high packetloss, some additional jitter. The benefit is the probability of loss andjitter is very low. The potential of lower latency and availability ofthe WAN service is very high.

With the APN memory services use of the APN services, an APN memoryrefresh may be transmitted using the APN load balancing services. TheAPN utilizes the quality reports from the WAN egress node and thecurrently queued packets awaiting transmission within the WAN ingressAPN node to do latency sensitive load balance. If the APN distributedmemory is of large size, transmitting the memory refresh generally mayrequire fragmentation. If the APN memory services were to transmit therefresh using only one path across the wide area network, then the APNmemory refresh would have a serialization delay and maximum transmitrate relative to the size of the distributed memory plus additionalprotocol overhead at the WAN ingress WAN link and at the WAN egress WANlink for the path. By utilizing the APN services, the memory regionfragments may be distributed in a time sensitive way optimized forsimultaneous arrival times at WAN egress utilizing multiple paths andmultiple WAN links. The cost is, because of the nature of the loadbalancing algorithm, some additional latency may be added. The benefitis that the overall serialization delay and transmission size is spreadacross multiple WAN links reducing the overall time across the network.

In brief, adaptive private network services utilize multiple independentdiverse networks, dynamic real time packet processing, universalsampling and analysis among its many aspects to increase the probabilityof having timely deterministic responses, aspects of a high qualityhighly responsive network, to be available to users of the adaptiveprivate network services.

FIG. 1 illustrates a processing complex 100 using synchronous sharedmemory for internal control of network adapters and various peripherals.The processing complex 100 includes a core processor unit 102, anexternal memory 104, and various peripheral adapter units 106 and 107and a network adapter unit 108 with a network interface 110. The coreprocessor unit 102 has a core memory complex 112, having one or morelevels of caching, a processing unit 114, having one or more processors,and a bus interconnect unit 116 for connecting to one or more adapterunits. The peripheral adapter units 106 and 107 and the network adapterunit 108 have an adapter memory 118, 120, and 122, respectively, as wellas an adapter processor and input/output (I/O) interface logic 119, 121,and 123, respectively. The adapter processors and I/O interface logic119, 121, and 123 may be implemented with an internal processor andmemory subsystem. Such adapter processors and I/O interface logic 119,121, and 123 and the processing unit 114 are responsive computerreadable media either downloaded or instantiated within the associatedunit. Also, as described in more detail below, an I/O unit 111 comprisesan adapter, such as network adapter unit 108 and an associated portionof bus interconnect unit 116.

The processor complex 100 also operates with a common clock 124 or withclocks derived from the common clock 124 such that operations on andbetween the core processor unit 102, the external memory 104, and thevarious peripheral adapters 106 and 107 and the network adapter 108 aresynchronous, have very high performance, and high reliability. Such aprocessor complex 100 may instantiate control operations and datatransfers through the use of shared memory operations between the coreprocessor unit 102 and each of the peripheral adapters 106 and 107 andthe network adapter 108. For example, the core memory complex 112 mayshare a block of memory locations 126 using communication paths 130-132with a corresponding block of shared memory locations (SM) 127-129. Dueto the high performance and high reliability of operations with theprocessor complex 100, updates to shared memory are synchronous withinthe time frame of multi-mega-hertz and giga-hertz clocks. A distributedshared memory organization facilitates programming of control operationsand data transfers and usually has low overhead due to the synchronousnature of the local communications. Extending a shared memoryarchitecture across a network is considered a very difficult problem dueto the unreliability of most network connections and the long latency ofcommunication between nodes in the network.

FIG. 2A illustrates a node processor complex 200 suitable for use as asender node or as a receiver node in an adaptive private network (APN)in accordance with the present invention. The node processor complex 200includes a processing unit 214, a memory complex 212 containing programmemory and data memory, external memory 204, input/output functions 211using a transport reliable protocol (TRP), and network interface 210.The methods described in connection with the embodiments disclosedherein may be embodied directly in one or more software modules executedby a processor and memory complex such as the node processor complex200, a personal computer, a server, or the like having one or morecentral processing unit devices. The processor node complex 200, forexample, may be configured to execute instructions under control of asoftware program stored on a computer readable storage medium eitherdirectly associated locally with the processor and memory complex, suchas may be available through an instruction cache, or accessible throughan I/O device.

FIG. 2B illustrates a network configuration 220 having a sender node 202and a plurality of receiver nodes 222 ₁, 222 ₂, . . . , 222 _(N) inaccordance with the present invention. The network configuration 220also includes a network 224 and routers 226 _(S), 226 ₁, 226 ₂, . . . ,226 _(N) for interconnecting the sender node 202 and the plurality ofreceiver nodes 222 ₁, 222 ₂, . . . , 222 _(N). The network 224 androuters 226 _(S), 226 ₁, 226 ₂, . . . , 226 _(N) collectively representan adaptive private network (APN) which may operate across the network224, such as the Internet. An adaptive private network (APN) may includewide area networks (WANs), adaptive private network appliances (APNappliances, or APNAs) in the network nodes, network applicationservices, as well as APN conduits between APN appliances. A WAN linkrepresents a physical access point to the wide area network (WAN), suchas a digital subscriber line (DSL) connection or a cable modem. One ofthe distinctive characteristics of a WAN link is bandwidth, representingthe amount of data capacity available for transmission and reception.WAN links may be shared among APN conduits, and intranet and Internetnetwork services. In the present embodiments, APN appliances do notdirectly attach to WAN links. APN appliances communicate with WAN linksthrough logical connections, such as provided by intermediate routers226 _(i) of FIG. 2B.

The sender node 202 includes a transmitting application 227, an adaptiveprivate network (APN) distributed transmit memory block 228, a memoryservices block 229, and a send/receive block 230 using a transportreliable protocol (TRP). Each receiver node “i” includes a receivingapplication 240 _(i), a receive memory block 241 _(i), a memory servicesfunction 242 _(i), and a send/receive function 243 _(i). Contents ofselected locations in the transmit memory block 228 may be shared andsynchronized with memory blocks in one or more of the receiver nodes 222₁, 222 ₂, . . . , 222 _(N) so that the remote receiver nodes have accessto the same data within a programmed update period or on an as neededbasis for more critical applications.

FIG. 2B illustrates the sender node 202 sharing a data set of a specifictype with receiver nodes 222 ₁ and 222 ₂. The memory services block 229is responsible for synchronizing the data set in the transmit memoryblock 228 of sender node 202 with corresponding data sets in receivememory blocks 241 ₁ and 241 ₂ of receiver nodes 222 ₁ and 222 ₂,respectively. The transmitting application 227 may write data to or readdata from the transmit memory block 228 over a memory interface 233. Thereceiving application 240 _(i) may only read from the receive memoryblocks 241 _(i) over a memory interface 248 _(i). In order to insure thetransmitting application 227 and receiving applications 240 ₁, 240 ₂, .. . , 240 _(N), maintain consistent data during a synchronizingtransmission, the data set being transferred is locked at each nodeduring the sending and receiving operations, as described in more detailbelow.

For example, memory updates in the receiver nodes 222 ₁ and 222 ₂ canoccur either as a result of an explicit update request received from thesender node, as shown in data path 223 shown in bold in FIG. 2B, or on aperiodic interval defined when services for a memory type areinitialized, or both. Whenever an update to a memory block is receivedin a receiver node “i”, an up-call 249 _(i) is called by the memoryservices function 242 _(i) to notify the receiving application 240 _(i)that an update has been received, and to perform whatever function isnecessary. Data transmitted is loaded into the receive memory block 241_(i) across a memory interface 247 _(i). Also, the memory servicesfunction 242 _(i) supports functions, such as, user defined compress anddecompress functions to compress or uncompress data sets for moreefficient reception and transmission across the network. Thecompress/decompress functions may also be used to update a data set in amemory block before sending or to construct a message to be sent, asdescribed in more detail below. The memory type is a value that isunique among communicating systems. The sender application and receiverapplications are coordinated to agree on the structure of a data set fora specified memory type.

FIG. 3 illustrates software communication layers in a sender node 202and a receiver node 222 _(i) using shared distributed memory inaccordance with the present invention. The sender node 202 includes thetransmitting application 227, the transmit memory block 228, the memoryservices block 229, and an APN send process 310, such as a transportreliable protocol (TRP), of send/receive block 230. The transmittingapplication 227 includes a compress function 306. The memory servicesblock 229 includes a shared memory send services block 302 and a sharedmemory send messaging block 304. The receiver node 222 _(i) includes thereceiving application 240 _(i), the receive memory block 241 _(i), thememory services function 242 _(i), and an APN receive process 330 _(i),such as a transport reliable protocol (TRP), of send/receive function243 _(i). The receiving application 240 _(i) includes a decompressfunction 326 _(i) and a memory refreshed indication service routine 327_(i). The memory services function 242 _(i) includes a shared memoryreceive services block 322 _(i) and a shared memory receive messagingblock 324 _(i).

On the sending side, the transmitting application 227 may be responsiblefor identifying destination nodes, identifying data sets to synchronizebetween nodes at a specified update period, setting of control blockdata, locking and unlocking data sets when doing an update process,requesting updates to be sent if necessary, and compressing sharedmemory data sets for message transmission. The shared memory sendservices block 302 is responsible for locking and unlocking data setswhen transmitting, updating communication timeliness, update ratethrottling to keep within a max number of updates within a specifiedtime period, and setting of a memory update version for tracking andindication. The shared memory send messaging block 304 is responsiblefor node-to-node messaging, data integrity checking, addresses,point-to-point transmission, point-to-multi-point transmission,fragmentation of large data sets prior to transmission, and updateversion checking.

On the receiving side, the receiving application 240 _(i) is responsiblefor decompressing shared memory data sets on message reception, readingof control block data as necessary, locking and unlocking data sets whenreading data from the receive memory block 241 _(i), and reacting toupdates as necessary. The shared memory receive services block 322 _(i)is responsible for locking and unlocking data when receiving data fromthe network 224, history logging, and placing an update up-call to thereceiving application 240 _(i). The shared memory receive messagingblock 324 _(i) is responsible for node-to-node messaging, data integritychecking, addresses, point-to-point reception, point-to-multi-pointreception, defragmentation of fragmented large data sets received duringtransmission, and update version checking.

FIGS. 4A-4D illustrates a shared memory block APN send process 400 inoperation at a sending node in accordance with the present invention.FIGS. 5A and 5B illustrates a shared memory block APN receive process500 in operation at a receiver node in accordance with the presentinvention.

The APN memory communicates between distributed components ofdistributed applications through APN-distributed memory regions. An APNmemory region is updated on the transmitting APN node, with the updatereflected via this instrument to one or more subscribing receivingnodes.

The use of the underlying APN network services are a significant factorin the operation of the APN distributed memory. Blocks 310 and 320 ofFIG. 3 are elements of the APN service that is utilized by the APNdistributed memory. The APN service provides for communication of datavia the APN conduit services. In addition, the APN service provides forhigh transmit priority of APN distributed memory data versus othertraffic via the quality of service (QoS) super critical priorityschedulers and shadows for WAN links. In addition the APN serviceprovides for higher throughput of APN distributed memory datatransmission via bandwidth allocation to higher quality paths avoidinglower quality paths. In addition, the APN service provides for deliveryreliability for APN distributed memory data via fast retransmissions onsingle packet loss detection. In addition, the APN service provides forreliability and timely delivery of APN distributed memory data throughredundancy transmissions via duplicate transmissions on high best pathand most independent path from best path.

A best available path process searches for an ideal set of paths havingunique ISP identifications on the remote and local WAN Links. A measureof independence of paths is determined such that one path may beselected having a greater measured level of independence than otheravailable communication paths. If an ideal set of paths is not found,the best available path algorithm searches for a set of paths havingunique ISP identifications on the remote WAN links and otherwise uniquelocal WAN links. Third, the best available path process searches forpaths with otherwise unique remote and local WAN links. Fourth, the bestavailable path process searches for paths with a single local WAN linkand otherwise unique remote WAN links. Should none of these sets ofpaths be found, the best available path process settles for one pathsharing a single pair of WAN links, which means the packets to beduplicated are simply transmitted on a single path twice. Once the bestavailable path or paths are determined, a packet buffer is allocated andthe packet is copied. Both packets are forwarded to a WAN link schedulerwhere the packets are scheduled on their respective WAN links.

The APN service also provides secure transmission of APN distributedmemory data using AES encryption. In addition, the APN service provideslow latency of APN distributed memory data transmission via a highresolution clock calibrated for use for high frequency path monitoringresulting in high latency path avoidance. Further details of the highresolution clock may be found in U.S. patent application Ser. No.12/482,766 filed Jun. 11, 2009 entitled “Flow-based Adaptive PrivateNetwork with Multiple WAN-Paths”, for example in FIGS. 1H, 1I, 2, and 7and at page 9, line 20 to page 18, line 17 and page 32, line 9 to page34, line 9. In addition, the APN service provides for low packet loss ofAPN distributed memory data via packet loss monitoring and loss pronepath avoidance. In addition, the APN service provides for low jitter ofAPN distributed memory data via path jitter monitoring and high jitterpath avoidance. In addition, the APN service provides for congestionavoidance of APN distributed memory data via high resolution clocksynchronized enabled congestion monitoring and avoidance.

The APN distributed memory service, as shown in block 229 and 242 ofFIG. 3, provides services to the transmitting and receiving applicationsas shown in blocks 227 and 240 _(i), respectively. The APN distributedmemory service provides for refresh of APN receive memory block 241 _(i)with data from the transmit memory block 228 of FIG. 3. A refreshoperation is a transaction for updating a receive memory and includestransmitting a block of memory having one to a plurality of data valuesto one or more receive memories in receiving nodes. Depending on thesize of a block of memory, the refresh operation may also includefragmentation into a plurality of data packets. A refresh operation mayalso be specified to use reliable transmission. In addition, the APNdistributed memory service provides for transmissions update priority asindicated via the transmitting application 227. In addition, the APNdistributed memory service provides for update priority as indicated bythe transmitting application 227. In addition, the APN distributedmemory service provides for fragmentation of transmit memory block 228on transmit side block 229 of FIG. 3 and reassembly of APN receivememory block 241 _(i) on the receive side memory services function 242.In addition, the APN distributed memory service provides for setting ofrefresh versions of transmit memory block 228 and version checking priorto accepting the refresh data and writing APN receive memory block 241k. The version checking prevents out of order older refresh data fromoverwriting data that is more recent. In addition, the APN distributedmemory service provides for the distributed application definedcompression function 306 and decompression function 326 _(i) fordistributed memory. In addition, the APN distributed memory serviceprovides for the locking and unlocking distributed memory to assure fullcoherency of an APN distributed memory refresh contents of the transmitmemory block 228 prior to transmit. The locking and unlocking ofdistributed memory is utilized to assure full coherency of an APNdistributed memory refresh contents of the receive memory block 241 _(i)to prevent a refresh from altering the receive memory block 241 _(i)when in use by the receiving application block 240 _(i). In addition,the APN distributed memory service provides for maximum threshold ofdistributed memory refreshes to be transmitted per second via means thatis not a linear time rate schedule within the period of the second. Inaddition, the APN distributed memory service provides for maximum timeduration permitted without a refresh transmit from the transmit memoryblock 228 of FIG. 3 and the receive memory block 241 _(i).

At time of initialization, the distributed applications on thetransmitting and receiving nodes, 202 and 222 _(i) of FIG. 3respectively, are expected to configure properties of the APN memoryservice for communication prior to utilizing the communication services.

The transmitting application 227 allocates blocks of memory, such as thetransmit memory block 228, that will be used for its distributedtransmit memory. The transmitting application 227 registers distributedtransmit memory to APN memory service 229 and is provided with areference to an APN memory service control block. The transmittingapplication 227 specifies desired properties of the APN distributedmemory service when invoking the registration routine. The transmittingapplication 227 may specify that a refresh should be scheduled with highpriority above other contending traffic using the conduit. Thetransmitting application 227 may specify that refresh should beduplicated using independent diverse paths within the conduit. Thetransmitting application 227 may also specify that the APN servicesshould retransmit a refresh packet if it is lost as it crosses thenetwork. The transmitting application 227 may also specify that the APNservices should be duplicated using independent diverse paths within theconduit. For example, the application may select duplication oftransmission or single transmissions. The application may also selectreliable retransmissions or no retransmissions when packet loss occurs.The application may also select super critical transmission with shadowscheduling or real time scheduling.

The transmitting application 227 then registers the remote APN nodesites that will be subscribed to the transmissions of the distributedmemory refreshes. Each remote APN node site must have an APN conduitconfigured to connect the node with a transmitting application to a nodewith a receiving application. The configured conduit will providesecurity, connectivity, path monitoring priorities, bandwidth and thelike as defined in U.S. patent application Ser. No. 12/482,766.

As shown in FIG. 3, at time of initialization, the transmittingapplication 227 allocates a block of memory, such as the transmit memoryblock 228, that will be used for the distributed transmit memory. Thetransmitting application 227 registers distributed transmit memory tothe APN memory service 229 and is provided with a reference to an APNmemory transmit service control block. The transmitting application 227associates additional properties to the APN memory services which arestored in the APN transmit service control block. A memory identifierproperty is specified which provides a unique identifier for the APNmemory element that will be used by the transmission and receiving nodesto identify the distributed memory binding. The memory identifier iscommon between the receiving application 240 _(i) and the transmittingapplication 227. The memory identifier is either well known prior toinitialization or is communicated between the transmitting and receivingapplication via other means.

A memory size property is specified which indicates the size of the boththe distributed transmit and receive memory blocks 228 and 241 _(i),respectively. This size is common between the receiving application 240_(i) and the transmitting application 227. The size is either well knownprior to initialization or is communicated between the transmitting andreceiving applications via other means. A maximum refreshes per secondthreshold property is specified which indicates the maximum number oftimes memory refresh will be permitted per second. A maximum durationpermitted without a refresh property is specified which indicates thelongest period of time permitted between memory refreshes. A compressionservice routine property may be specified that references a compressfunction 306 that is invoked prior to any transmission of thedistributed memory for compression of the contents to a potentiallysmaller size for transmission. If no compression service routine isspecified, the APN memory service transmits the APN memory unmodified.One suitable means of compression is very highly correlated to theapplication type and the data that may be contained in the memory. Ageneral purpose application may not be optimal in all cases, where anapplication defined service routine is able to utilize greateroptimization assumptions that are specific to the application. Forexample, video compression may provide means of compressing key framesoccasionally, and providing delta updates between refreshes, whereastext data may use simple Lempel-Ziv-Welch compression methods.

As shown in FIG. 3, the receiving application 240 _(i) allocates a blockof memory that will be used for its distributed receive memory block 241_(i). The receiving application 240 _(i) registers the distributedreceive memory to the APN memory service function 242 _(i) and isprovided with a reference to the APN shared memory receive servicesblock 322 _(i). The receiving application 240 _(i) associates additionalproperties to the APN memory services which are stored in the APNreceive service control block. A memory identifier property is specifiedwhich provides a unique identifier for the APN memory element that willbe used by the transmission and receiving nodes to identify thedistributed memory binding. The memory identifier is common between thereceiving application and the sending application. The memory identifieris either well known prior to initialization or is communicated betweenthe transmitting and receiving applications via other means. A memorysize property is specified which indicates the size of both thedistributed transmit and receive memories. This size is common betweenthe receiving application and the sending application. The size iseither well known prior to initialization or is communicated between thetransmitting and receiving applications via other means. A set of remoteAPN node sites identifiers are specified to identify nodes that arepermitted to transmit to the specified distributed receive memory.

A decompression function 326 _(i) may be invoked upon reception of amemory refresh for its decompression. If specified, the receivingapplication's decompression function is configured to be compatible withthe transmitting application's compression function. If specified, thedecompression function 326 _(i) is invoked prior to invoking anyspecified memory refreshed service indication routine. A memoryrefreshed indication service routine 327 _(i) may be invoked wheneverthe distributed receive memory is updated by the APN memory service.

The APN distributed memory is configured to be unidirectional inoperation. The APN distributed transmit memory block 228 is written bythe transmitting application. The APN distributed receive memory block241 _(i) is read by the receiving application. If a distributedapplication requires bidirectional communication, then twounidirectional distributed memories pairs may be used, one memory regionfor communication in one direction, the other memory region forcommunication in the other direction.

A local locking facility is used at the sender node 202 and the receivernode 222 _(i) for both the APN distributed memories to preserve anatomic snapshot of a memory refresh in order to prevent incompleteupdates. The locks are done via a locking mechanism in hardware or via asoftware operating system variation such as a variation of a pthreadlibrary mutex facility that utilizes a multiprocessor spinlock.

It should be noted that a lock is between the transmitting or receivingapplications and the APN memory services at each node. The lock is not anetwork wide lock between the transmitting application and the receivingapplication. Since a single instance of memory is never written to byboth the transmitting and the receiving applications there is no need toprevent conflicting writes between the transmitting and receivingapplications by using a network wide lock. The transmitting applicationsand the receiving applications are asynchronous to each other and, ifused properly, should not require a network wide lock. Network widelocks would have substantial issues if attempted. For example, if node Alocked a local memory from access by node B, node B would be required towait for the lock to be removed before node B could write data to nodeA's local memory. If node A powered down prior to releasing the lock onits local memory, node B would potentially have an undetermined wait forthe lock to be cleared, if ever.

The transmitting application may update its distributed transmit memorywhen needed. FIG. 4A illustrates an exemplary process 400 suitable foruse in transmitting application 227 of FIG. 3. In FIG. 4A, if the updateis not atomic or multiple changes within the distributed memory blockare interdependent, then a lock must be performed on the memory prior tothe update, and then unlocked afterward. If there are changes todifferent data elements that are not interdependent, then a lock is notrequired. When a lock is acquired, the APN memory service is preventedfrom sending an APN memory refresh to the receiving node until the lockis freed. At block 401, if the transmitting application wishes to do anon-atomic update to the distributed transmit memory, the transmittingapplication will attempt a lock. At block 402, the APN memory servicewill evaluate the APN memory service control block to determine if alock may be granted. If the lock is not able to be granted, thetransmitting application reattempts the lock at a later time. When thelock is granted, the transmitting application may proceed to update thedistributed transmit memory at block 403. During the period the lock iscontrolled by the transmitting application, the APN memory services willnot be able transmit refreshes to the receiving node so this periodshould be optimized to be as short as is practical. At block 404, whenthe update is completed, the transmit application unlocks the APN memoryso that the region may be accessed by the APN memory service or otherprocess entities contained within the APN transmitting application thatare pending for an update to the APN distributed transmit memory.

For example, if the application is doing a simple posted write of asmall size data element, such as writing an application context statebyte field to the distributed transmit memory, then a lock is not neededsince the write of the data element is atomic. Similarly, if an updatehas multiple data elements that the distributed application does notview as interdependent, then a lock is not needed. An example of thismay be a simple network management protocol (SNMP) with a managementinformation base (MIB) structure where updating the statistics of eachelement within the MIB structure does not typically have crossdependencies. In both these cases, there is no potential for an APNmemory refresh to be transmitted to the receiving application with onlya partial change. On the other hand, if the APN distributed transmitmemory was to contain two data elements that were interdependent, then alock would be required. For example, if one data element identified apixel location and another data element identified the color of thepixel, then these data elements are codependent and a lock should beused for any update to either data element. If a lock was not acquiredprior to such a write, there is the potential that the APN memoryservice may attempt to transmit a refresh after the write of the firstelement but before the write of the second element was completed by thetransmitting application. The result of such an occurrence would be thatthe receiving application would have an incomplete partial refresh thatcould have negative ramification.

Similarly, the receiving application may need to lock the APN receivememory block if an access to multiple data elements within areinterdependent. The APN distributed receive memory at the receiver nodeis volatile in that it may be altered by the APN memory process at anytime when not under local lock. If a receiving application needs to havean extended period to process data contained within APN distributedreceive memory block that has interdependent elements, it may not bepractical to do this under lock. In this case, the receiving applicationapplies a local lock on the APN distributed receive memory block, copythe APN distributed receive memory block to a private memory, and thenunlock the APN distributed receive memory. The receiving applicationshould not write to APN distributed receive memory block. Such a writecould result with data being lost by a future asynchronous writing ofthe APN distributed receive memory by the APN memory services as aresult of a reception of an APN memory refresh received from thetransmitting node.

The APN memory transmitting application polls for transmit refreshrequests periods. The APN shared memory send service 302 at thetransmitting node has, for example, three polling cycles to providerefresh opportunities to registered distributed memories. These pollcycles may be set at 100 ms, 10 ms, and an event poll at a highfrequency. The high frequency poll may be at differing intervalsdependent on the underlying operating system and hardware platform. Inthe current implementation, an event poll is called after each packetreception is complete within the APN service.

The APN memory transmitting application provides an indication ofrefresh request priority. When the transmitting application 227 updatesan APN distributed transmit memory 228, the transmitting application 227may optionally request an immediate refresh be sent out to subscribingnodes. If no update indication is signaled to the APN shared memory sendservice 302 by the transmitting application, the distributed memory isrefreshed at a next request or at an expiration of a maximum durationpermitted without a refresh time period, whichever is first. An exampleof a transmitting application not requiring an explicitly requestedrefresh of an APN memory is, for example, writes to simple counters forpacket counts that do not require high frequency of refresh at receivingnodes. For this type of network statistics, a once per second refreshinterval is satisfactory.

If the transmitting application has time sensitive data that must betransmitted faster than a maximum refresh threshold duration, then theapplication may indicate one of three explicit priorities. Waiting for amaximum refresh threshold duration is considered to be a lowest priorityAPN memory transmission indication and is implied if no other APN memorytransmission indication is specified. An APN memory timely transmissionindication is considered to be a next lowest priority for APN memorytransmission after the implied maximum refresh threshold duration. Arequest for APN memory timely transmission indication is used if thetransmitting application requests an APN memory refresh at the next 100millisecond (ms) interval or next opportunity satisfied by higherpriority request prior to the next 100 millisecond interval. An APNmemory critical transmission indication is considered to be the nextlowest priority APN memory transmission after the APN memory timelytransmission indication. A request for APN memory critical transmissionis used if the transmitting application requests an APN memory refreshat a next 10 millisecond interval or at a next opportunity satisfied byhigher priority request prior to the next 10 millisecond interval. APNmemory immediate transmission indication is a highest priority request.A request for APN memory immediate transmission is indicated if thetransmitting application requests an APN memory refresh at a next highresolution poll opportunity.

If multiple requests are initiated prior to the next poll opportunity,the requests are combined into a single refresh to prevent needlessredundant updates. For example, if a timely request is requested, and acritical request is requested, then both requests will be satisfied witha single transmit at the next 10 ms poll. When an APN refresh isperformed, the maximum refresh threshold duration period timer isrestarted resulting in a period ending in the future timed from the lastrefresh regardless of what indication priority that refresh was.

A pending transmit priority may be stored in a bitmask field to enablethe APN memory services to be more efficient when polling. Withmultithreaded and multicore implementations, one threaded processor orone multicore processor of the transmitting application may attempt toread, then modify, then write a pending transmit priority bit whileanother threaded processor or multicore processor of the transmittingapplication is doing the same. This may result in a data hazard where arefresh priority request is inadvertently lost. As a result, process 414of FIG. 4B may be used as an exemplary means of locking the bitmaskfield prior to setting a request priority. FIG. 4B illustrates theexemplary process 414 suitable for use in shared memory send servicesblock 302 of FIG. 3. At block 415, if the transmitting applicationwishes to transmit a memory location from the distributed transmitmemory to remote users, the transmitting application attempts a lock onthe memory location. At block 416, the APN memory service evaluates theAPN memory service control block to determine if a lock may be granted.If the lock is not able to be granted, the transmitting applicationreattempts the lock at a later time. When the lock is granted, thetransmitting application sets transmit request flags and sets a priorityfor transmission at block 417. At block 418, after setting thetransmission request flags and setting the priority, the transmittingapplication unlocks the memory location so that it may be accessed bythe APN memory service or other process entities contained within theAPN transmitting application that are pending for an update to the APNdistributed transmit memory.

The transmitting application may be configured to provide multiplefrequency thresholds. For example, a maximum duration between updatesthreshold assures that a receive memory block is updated at thesubscribing nodes at no less than once per that duration. If an updatetakes place because of a refresh request and poll prior to the durationexpiring, the refresh duration time start time stamp is retaken and thenext interval will take place no longer than that maximum duration inthe future from that poll transmit.

A maximum threshold of APN memory refreshes per second is an upper limitof refresh occurrences per second permitted for the APN memory. Thisthreshold does not specify a rate for a scheduler, in that allocatedrefreshes are not evenly distributed, but rather is treated as a limit.For example, if it was specified that the memory may be updated no morethan 5 times a second, all 5 updates may happen within 10 millisecondsfollowed by 990 milliseconds of no updates. This is achieved by atransmit time table that tracks a cumulative number of refresh timeswhich may be compared to “n” representing a maximum number of refreshespermitted. If a delta of the current requested refresh time and thecurrent—nth refresh is less than one second, the transmission isdeferred until the delta exceeds one second. The purpose of this method,versus a rate scheduler method, is to provide the timeliest updates whenneeded and use the upper threshold only as an occasional limit. If arate schedule was used, every update may be delayed by a certain period.This would result in a delay for typical and best cases. By comparing toa maximum number of refreshes limit, updates are allowed to take placeat a much higher frequency for short durations without delay for thetypical or best cases, but at a cost associated with a worse case delay.For most time sensitive applications, this is a very good trade offoptimizing for best and typical timeliness.

FIG. 4C illustrates an exemplary process 428 suitable for use in theshared memory send messaging block 304 of FIG. 3. At block 430, aprocess of transmission of an APN memory block via an APN memory pollloop is started. At block 431, a determination is made whether there aretransmission request entries on a pending transmission list. If thereare no pending transmission requests, the process 428 returns to theinitiating routine at block 432. If there are pending transmissionrequests, the process 428 proceeds to block 433. At block 433, a memoryblock associated with the next entry in the pending list is allocatedfor transmission. At block 434, a determination is made whether atransmission request flag is set and the request has appropriatepriority for this transmission. If the transmit request flag is not setor the priority is not proper, the process 428 returns to block 431. Ifthe transmission request flag is set and the priority is proper, theprocess 428 proceeds to block 435. At block 435, a determination is madewhether a frequency threshold has been exceeded. If the frequencythreshold has been exceeded, the process 428 returns to block 431. Ifthe frequency threshold has not been exceeded, the process 428 proceedsto block 436. At block 436, a determination is made whether theapplication requests the memory block be compressed. If the memory blockis to be compressed, the process 428 proceeds to block 437. At block437, the compress function 306 is executed and the compressed memoryblock is saved in a temporary buffer under lock control. At block 442,the temporary buffer is transmitted using the APN send process 310 anddescribed in more detail with regard to FIG. 4D below. After initiatingthe transmission, the process 428 returns to block 431.

Returning to block 436, if the memory block is not to be compressed, theprocess 428 proceeds to block 438. At block 438, the memory locationassociated with the allocated memory block is checked to see if itlocked. At block 439, a determination is made whether the attempt madein block 438 indicates the allocated memory is locked. If it is not yetlocked, the process 428 returns to block 438. If the allocated memory islocked, the process 428 proceeds to block 440. At block 440, theallocated memory is copied to a temporary buffer. At block 441, theallocated memory location is unlocked. At block 442, the temporarybuffer is transmitted using the APN send process 310 and as described inmore detail with regard to FIG. 4D below. The process 428 then returnsto block 431.

FIG. 4D illustrates an exemplary process 458 suitable for use in the APNsend process block 310 and associated with the shared memory sendmessaging block 304 of FIG. 3. At block 460, a process of transmissionof APN memory blocks via the APN memory poll loop is started. At block461, a decision is made whether there are available transmissionconduits on a subscribed list of conduits. If there are no availabletransmission conduits, the process 458 returns to the initiating routineat block 462. If there are available transmission conduits, the process458 proceeds to block 463. At block 463, the process 458 selects a nextavailable conduit. At block 464, a determination is made whether therequested block of memory is larger than the selected conduittransmission size. If the requested block of memory is larger than theselected conduit transmission size, the process 458 proceeds to block465. At block 465, the block of memory is fragmented into multipletransmission blocks to fit within the conduit maximum transmission size.Further details on fragmentation may be found in U.S. patent applicationSer. No. 12/482,766 filed Jun. 11, 2009 entitled “Flow-based AdaptivePrivate Network with Multiple WAN-Paths”, for example in FIGS. 1H, 3A-C,and 4 and at page 9, line 20 to page 11, line 2, at page 21, line 18 topage 30, line 5, and at page 45, line 11 to page 46, line 7. The process458 then proceeds to block 467. Returning to block 464, if the requestedblock of memory is not larger than the selected conduit transmissionsize, the process 458 proceeds to block 466. At block 466, the number oftransmission blocks is set to one. At block 467, a determination is madewhether there are more blocks to transmit. If there are no more blocksto transmit, the process 458 returns to block 461. If there are moreblocks to transmit, such as the single block according to block 466, theprocess 458 proceeds to block 468. At block 468, a determination is madewhether the transmission application has requested reliabletransmission. If the transmission application has requested reliabletransmission, the process 458 proceeds to block 469. At block 469, theblock of memory is marked for reliable APN transmission and the process458 proceeds to block 470. Returning to block 468, if reliabletransmission has not been requested, the process 458 proceeds to block470. At block 470, the block of memory is transmitted on the selectedconduit APN reference path, as coordinated and controlled by the APNsend process 310. Further details of APN transmission may be found inU.S. patent application Ser. No. 12/482,766 filed Jun. 11, 2009 entitled“Flow-based Adaptive Private Network with Multiple WAN-Paths”, forexample in FIGS. 1H-J, 2, and 3A-C and at page 9, line 20 to page 18,line 17 and at page 21, line 18 to page 36, line 20. At block 471, adetermination is made whether the transmitting application is requestingduplication. If the transmitting application is not requestingduplication, the process 458 returns to block 467. If the transmittingapplication is requesting duplication, the process 458 proceeds to block472. At block 472, the block of memory is also transmitted on a bestavailable path that is most independent from the selected conduitreference path and then the process 458 returns to block 467. Furtherdetails of duplicate transmission may be found in U.S. patentapplication Ser. No. 12/482,766 filed Jun. 11, 2009 entitled “Flow-basedAdaptive Private Network with Multiple WAN-Paths”, for example in FIG.3C and at page 31, line 18 to page 36, line 20.

This section addresses the APN memory services use of versioning for APNmemory refreshes. Each APN memory refresh update is marked with anappropriate version at the transmission node in the order the refreshesare transmitted by the APN transmitting node. A version tag may include,for example, a time stamp or, alternatively, a version number that isincremented with each new update. The receiving node tracks the updateversions it receives. If the receiving node receives an update that isof earlier version, the update is ignored. The APN memory facility isconfigured to operate based on a rule that memory refreshes arecumulative. Any one APN memory refresh may be lost if the update wasreceived out of order. The APN memory facility is ideal for real timestatus and statistics updating.

FIG. 5A illustrates an exemplary process 500 suitable for use in theshared memory receive messaging block 324 _(i) in a receiver node 222_(i) of FIG. 3. At block 501, an upcall is received for a frame from theAPN receive process 330 _(i). At block 503, a received message is parsedto determine a source site and message identification (ID). At block504, a control block is selected from database 510 based on the receivedsource site and message ID appropriate to receive a memory blocktransmission. At block 505, a determination is made whether the messageversion is greater than the last accepted version. If the messageversion is not greater than the last accepted version the message is aduplicate message and the process can be stopped at block 502. If themessage version is greater than the last accepted version, the process500 proceeds to block 506. At block 506, the received message data iscopied into a fragmentation buffer and if needed reassembled. At block507, a determination is made if the buffer is complete indicating allfragments of the memory block have been received. If all of thefragments have not been received, the process 500 stops at block 502,until the next fragment is received. If all of the fragments have beenreceived, the process 500 proceeds to block 508. At block 508, adetermination is made whether a checksum operation on the data in thebuffer indicates no errors. If errors are found the process 500 stops atblock 502. If there are no errors, the process 500 proceeds to block509. At block 509, an upcall is placed to shared memory receive servicesblock 322 _(i) for receive handling of the memory block transmitted.

FIG. 5B illustrates an exemplary process 548 suitable for use in theshared memory receive services block 322 _(i) in a receiver node 222_(i) of FIG. 3. At block 550, the upcall from block 509, shared memoryreceive messaging block 324 _(i), is received in the shared memoryreceive services block 322 _(i). At block 551, the upcall is registeredand the decompress function 326 _(i) is called. For example, thedecompress function 326 _(i) may also be responsible for locking. Atblock 552, a determination is made whether there is a memory refreshedservice indication routine to be invoked. If there is no memoryrefreshed service indication routine, the process 548 stops at block554. If there is an appropriate memory refreshed service indicationroutine, the process 548 proceeds to block 553. At block 553, a receivememory update handler in receiving application 240 i is called. If thememory refreshed service indication routine's access to the distributedreceive memory requires the memory to be non-volatile to changes bysubsequent memory refreshes during the period the service routine isprocessing, the service routine will be responsible for locking thedistributed receive memory as noted prior for the receiving applicationuse in general. The process 548 stops at block 554.

FIG. 6 illustrates a high availability (HA) network configuration 600having an active node 602 _(A), a standby node 602 _(S), and a pluralityof receiver nodes 222 ₁, 222 ₂, . . . , 222 _(N) using send/receivetransport reliable protocol (TRP) blocks 630 _(A), 630 _(S), 243 ₁, 243₂, . . . , and 243 _(N) in accordance with the present invention. A highavailability site 601 deploys two nodes with one node configured tooperate in an active state, the active node 602 _(A), and the other nodeconfigured to operate in a standby state, the standby node 602 _(S). Theactive node 602 _(A) performs all node functions, such as describedabove with regard to node 202 of FIG. 2B. The standby node 602 _(S)monitors the state of the active node 602 _(A) and in the event of afailure of the active node 602 _(A) takes over the active nodeoperations. The distributed shared memory services of the presentinvention are used by the active node 602 _(A) and the standby node 602_(S) to communicate their operational status information to each other.A redundant control protocol (RCP) state machine operates in both activeand standby nodes to communicate between the redundant nodes for thepurposes of exchanging information necessary to elect the active nodeand monitor the health of both nodes. Such exchange of information isfacilitated by use of point-to-point memory synchronizationcommunicating a common memory type. The send/receive blocks 630 _(A),630 _(S), 243 ₁, 243 ₂, . . . , and 243 _(N) provide reliabletransmission across a network, control the transmitting of packets andretransmitting of packets as required, controls the receiving andacknowledging of packets, and supports packet duplication for reliabletransmission.

A detailed use case of the APN memory facility is described includinghow it uses the unique underlying APN services as well as how it is usedby the APN services.

The operation of the adaptive private network uses the APN nodes at WANegress to monitor and analyze all traffic as it is transmitted andreceived across the different wide area networks for latency, jitter,loss, and congestion for each path. The WAN egress nodes of the APNcommunicate to the WAN ingress nodes the results of the analysis so thatthe WAN ingress nodes may use the information to direct traffic acrossthe network in a highly reliable timely way. The reports of the networkbehavior are referred to as APN quality reports, or conduit qualityreports. Included within the quality reports are individual qualityreports for each path associated with the conduit which are referred toas path quality reports, information on each WAN link, referred to asWAN link quality reports, and information on the conduit as a whole.Information in the path quality report includes counts of packetreceived, packets lost, bytes received, bytes lost, operational stateindication, latency, and jitter. Information in the WAN link qualityreport includes bytes received, bandwidth permitted, and congestionindication. Information within the conduit quality report for theconduit as a whole contains maximum transmission unit permitted, lowestlatency observed, and bandwidth permitted.

If the quality reports were not propagated from the WAN egress node tothe WAN ingress node in a timely and reliable way, the APN would notgenerally be able to adapt and converge to rapid changes in the widearea networks that the APN is utilizing for communication. When the APNnode propagates a quality report to its peer, the APN node updates thedistributed transmit memory with the current analysis of the conduit'spaths and their WAN links.

The APN node may indicate to the APN memory services a refresh requestpriority. The priority of the refresh request is highly correlated tothe type of information that is being propagated. For example, if theAPN node is updating packet receive counters, the APN memory refresh isnot considered to be time critical. For this use, the APN node generallydoes not issue any refresh request priority indication. The refresh isissued by the APN memory services after waiting for the maximum refreshthreshold duration to expire or if the refresh is otherwise satisfied byhigher priority request prior to the maximum refresh threshold durationexpiring. On the other hand, if the information to be propagated is of ahigher priority, such as the detection that a path has becomenon-operational, the refresh request priority is generally set to animmediate transmission indication. This results in the APN memoryrefresh being transmitted at the very next opportunity. The result isthe APN memory refresh is received by the APN node at the WAN ingresslocation and permits the node to divert traffic away from thenon-operational path faster.

The quality reports contain data about various paths and WAN links. Thedata in the quality report memory is generally stored in a sparse formatthat is optimized for fast table record access and retrieval viaindexing into fixed offset in the memory region. Although efficient forprocessing time this comes at a cost of additional space needed for thequality reports storage in memory. This is compared to a potential morepacked format which would utilizes less memory but require more indirectaccesses to records via a hash table of alternative data structures, forexample. The choice in the present implementation is to use a sparseformat in memory since that is more efficient for packet processingroutines to access. The consequence of this choice could be a very largedistributed memory that will consume substantial amount of networkbandwidth as memory refreshes are propagated across the network. Asnoted prior, the quality reports are transmitted using the APN duplicateand reliable services, so the bandwidth consumed is greater as a result.To provide a more optimal solution, the APN utilizes a pack and anunpack utility as its APN memory compression and decompression serviceroutines. The pack routine at the transmitting node takes the sparselypopulated APN quality report memory and restructures the data layoutinto a much more space and transmission efficient tightly packed format.The decompression service routine on the receiving node unpacks thememory refresh and returns it to its un-packet less efficient for spaceand transmission but more efficient for processing a sparse format.

Even though the packed format of the quality reports is substantiallysmaller than they would be if transmitted unpacked, the size of thequality reports typically exceed a maximum transmission unit permittedover the wide area networks available. In this case, the memory refreshassociated with the quality report is fragmented and transmitted acrossthe WAN network on one or more paths between the APN nodes. At thereceiving APN node the APN refresh is reassembled from the fragments.The benefit is the aggregated bandwidth of the available wide areanetworks are able to be utilized for the quality reports.

The quality reports contain within them WAN link quality reports.Elements of these quality reports contain information pertaining to themaximum transmission rate permitted for various WAN links. Uponreception at the WAN ingress APN node, the traffic schedules containedfor each WAN link may need to be adjusted as a result of the new dataavailable in the quality report. When the APN memory service receives amemory refresh and has defragmented and unpacked, decompressed, therefresh, the APN memory services invokes the memory refreshed serviceindication routine. Contained within the service routine is logic thatlocks the distributed receive memory and then adjust the schedulers tomatch the permitted rate requested, and then unlock the distributedreceive memory and then ends the service routine.

A level of congestion of a WAN link may be detected by constantmonitoring of the arrival rate of messages from the WAN link, accountingfor data and periods of no data, referred to as holes versus anidealized anticipation of a model of the WAN link's ability to receiveat WAN egress node.

One approach to controlling the level of congestion in accordance withthe present invention is through the use of a network clock in frametagging and rate comparison. For example, 32 bit network time timestampsof microsecond (us) granularity, for example, may be used. The networktime is a high resolution clock sync maintained across the network, asfurther addressed in U.S. application Ser. No. 12/482,766 incorporatedby reference herein in its entirety. By using the 32 bit time stamp, thesending WAN ingress node and receiving WAN egress node can use areference clock of estimated microsecond granularity. The 32-bit timestamp provides improved granularity of control compared to a 16-bit timestamp for example. It is appreciated that other levels of granularitymay be used.

This does not mean that the clock sync is of microsecondsynchronization. It only means that the clock is used to estimate thenetwork time in microseconds. If there is a drift of up to amillisecond, this error would be quickly resolved because of the clockcalibration algorithm and any impact of a temporary drift would beminimal to this algorithm.

The current invention alters the WAN ingress scheduler's rate, via themeans documented in U.S. application Ser. No. 12/482,766, for example.The use of rate changes on WAN egress may be utilized to detect, toadjust, and to more precisely measure congestion in many situations.

In accordance with the present invention, congestion is tracked on eachindependent path from WAN ingress to WAN egress. Rather than having onetracking control block for each egress WAN link, each path is trackedindependently. The packet stream applies a more refined filter thatprevents loss of data which may be considered significant. Whendetecting if congestion is occurring, all the paths are analyzed.

An exemplary 100 ms period is used in a running time sample table, toanalyze congestion. By summing into 100 ms samples, and then utilizing atime sample table, we can compare the behavior vs. fixed time periods.In the current implementation, the sample table contains 10 samples withtheir duration included with each sample. Whenever used, the summationvalue is adjusted to be a rate of change per second value.

An exemplary approach to dealing with congestion and to minimize itseffects is as follows.

Packet reception at the congestion detection algorithm:

-   1. A packet is received, it's received time in local clocks is noted-   2. The path context control block is fetched for the packet-   3. the packet's receive time in microseconds (us) is converted to    network time in microseconds-   4. the packet's transmit time in network time is exacted from the    packet-   5. The packets size, converted to account for WAN link overhead and    quantum is added to path running count of bytes received.-   6. if the packet is the first packet received,    -   a. packets receive time in local time us is recorded    -   b. the transmit time in network time us is recorded    -   c. the current running count of bytes is recorded in the current        sample    -   d. the subroutine exits-   7. the send time delta network time us is calculate by delta between    the last recoded send time in network time us and this packet's send    time in network time us is calculated.-   8. The receive time delta us is calculated by delta between the last    recoded receive time in local us, converted to network time us, and    this packet's receive time in network time us is calculated. The    storing of arrival times in local time and converting them later to    network time allows for the network time to be continually adjusted    after the recording. It is best to convert local time to network    time at the last possible instant before its use since a new the    network clock is always better than an older one when used at the    present.-   9. If the send time delta network time us is a negative number, the    packet has arrived out of order.    -   a. The subroutine exits and the packet is not added to the        sample period.-   10. The send rate is calculated by dividing 1 million by the send    time delta in network us.-   11. The receive rate is calculated by dividing 1 million by the    receive time delta in network us.-   12. The send rate and receive rates are converted to Kbits per    second resolution with double floating point precision.-   13. If the send rate or receive rate are above the physical limits    of the WAN link,    -   a. The subroutine exits and the packet is not added to the        sample period. On the WAN egress side, this can happen if, after        the WAN egress WAN link, a burst of packets is sent to the node        because of temporary delay at an intermediate device. This means        that we have lost any ability to use this data, since it was        jittered post WAN link. On the send side, the APN will        occasional schedule critical packets faster than the WAN link        configured rate. If used, these could result in perception of        congestion when actually the WAN link was doing proper        scheduling.-   14. If the send delta times or the receive delta times are of    extreme duration, the packet is treated as the first packet of the    flow. This could be the result of the remote node being reset or    even replaced.    -   a. packets receive time in local time us is recorded    -   b. the transmit time in network time us is recorded    -   c. the current running count of bytes is recorded in the current        sample    -   d. the subroutine exits-   15. send_vs_receive_delta_nus is calculated by subtracting the    receive delta time network us (nus) from the send delta time nus.    This shows the instantaneous trend of the congestion. If the packet    is received with less delta than the send, congestion is being    reduced. If the packet has more congestion, congestion may be    occurring. This also shows jitter in the networking as well, so the    system does not over react to a single instance.-   16. If the packet is a valid sample, the send_vs_receive_delta_nus    value is added into the current samples running for congestion.-   17. If the sample duration is over 100 ms,    -   a. the sample is added into the long duration sample table for        this path    -   b. the sample is reinitialized to start a new sample period-   18. Return    An exemplary approach to 100 ms poll congestion detection is as    follows:-   1. for all the paths for a WAN link from a particular conduit    -   1. if the path has received samples        -   a. Calculate the rate of change per networking us over the            period covered by the sample table using double precision            floating point. Adjust the rate of change to be rate of            change over a second.        -   b. If the negative rate of change in network time is greater            than a configure threshold, congestion is detected on this            path. In the current implementation, the default setting is            −20 ms per second rate of change between receive rate and            send rate but is configurable.-   2. If all operational paths for a conduit are indicating congestion,    the WAN link is considered congested. If any operational path for a    conduit is not congested, the WAN link is not considers congested.-   3. WAN link congestion will not be indicated for the WAN link if no    path using the WAN link has been sampling for greater 500 ms.-   4. Return    An exemplary approach to 100 ms poll for WAN link congestion    mitigation is as follows:-   1. Calculate the measured receive rate for all paths on a conduit    for this 100 ms period.-   2. Adjust the calculated rate to Kbits per second using double    floating point precision.    An exemplary approach to congestion exit detection is as follows:-   1. If WAN link in congestion monitoring state    -   a. if        -   i. congestion was not detected in current sample        -   ii. And no congestion has been detected for 200 ms        -   iii. And the WAN link received rate is 90% of the permitted            congestion flag for WAN link            -   1. WAN link is no longer in congested state and                congestion rate adjustment do not need to be done                An exemplary approach to congestion rate adjustment is                as follows:-   1. if WAN link congested just detected in this 100 ms period    -   a. Set rate factor to 80%-   2. else if last 100 ms WAN link received rate is greater than 120%    of permitted rate    -   a. set rate factor to 90%, since this shows we are gradually        reducing congestion-   3. else If last 100 ms WAN link received rate is greater than 105%    of permitted rate    -   a. set rate factor to 100%, since this shows we are have reduced        congestion-   4. else    -   a. Set rate factor to 110% since this shows he rate has been        reduced below our configured scheduled rate.-   5. Adjust the permitted rate to be the last 100 ms measured received    rate multiplied by the rate factor-   6. Signal the rate adjustment to the WAN ingress node

When congestion is detected on a WAN link, the permitted rate the WANingress node is allowed to transmit to the WAN egress node via thecongested WAN link is adjusted up to every 100 ms based on measurementof the receive rates of the network versus the previously prescribedpermitted transmit rate. When the measured received rate is betweenwithin 105% and 110% of the permitted rate, congestion is beingmitigated so no adjustment is done. When the measured rate issignificantly above the permitted rate (120%), the congestion is beingdrained, but we have a substantial backlog so the rate is adjusted downin 10% increments against the current measured rate level per 100 msperiod. When the measured received rate is significantly below thepermitted rate, congestion may be letting up or the WAN ingress node nolonger has much data to send, so increases of the permitted rate areallowed in 10% increments per 100 ms period against the current measuredrate.

Assuming a WAN link is used that is 100 kbps in size, in whichcongestion has been detected, and the measured received rate is 50 kbps.Initially, the permitted rate of the WAN link is reduced to 80% of thelast sampled received rate of the WAN link. For this example, theadjustment would set the new permitted rate to (0.8*50 kbps) 40 kbps. Ifcongestion is detected in future 100 ms checks, the permitted rate iscontinued to be adjusted. If in subsequent 100 ms checks, the measuredrate stays significantly above the current permitted rate, the algorithmstill incrementally reduces the permitted rate to 90% of the measuredreceived rate for that 100 ms. If a receive rate is measured to be 45kbbps, the rate would then be adjusted down to 40.5 kbps. If the receivemeasured data rate is below the permitted rate, then the congestion maybe easing so the current permitted rate may be increased. If themeasured data rate falls to 30 kbps, then the current permitted ratewould be increased by 110% of measured rate, or, for example, 33 kbps.It is possible this improvement in data rate was caused by thecongestion situation being eliminated or by WAN ingress node no longerhaving substantial data to send to the WAN link.

In practice, under congestion mode operation, the rate adjustments willcycle the rates up and down about the rate at which an actual congestedlink is abile to transmit data without substantial delay until theincident that caused the congestion is resolved.

Among its several aspects, this invention reduces the effect of networkbandwidth, packet loss, congestion, jitter, and latency, from being afactor for highly responsive distributed applications across the publicInternet. The invention also greatly increases the probability for itssuccessful application across a greater number of typical networks andapplications uses where timeliness and responsiveness are required.While the present invention has been disclosed in the context of variousaspects of presently preferred embodiments, it will be recognized thatthe invention may be suitably applied to other environments consistentwith the claims which follow.

We claim:
 1. A method of sharing memory in a network having multiplecommunication paths between distributed memory nodes, the methodcomprising: fragmenting a block of memory having a transmission sizeexceeding a transmission capacity of a first communication path into afirst fragment and a second fragment; transmitting the first fragmentfrom a transmitting node across the first communication path to areceiving node and transmitting the second fragment from thetransmitting node across a second communication path to the receivingnode, the second communication path selected from multiple communicationpaths as having a highest measure of independence among the multiplecommunication paths; receiving the first fragment and the secondfragment in the receiving node as a combined memory block; and storingthe combined memory block at a location in a distributed memory in thereceiving node, the location locked to assure full coherency of thecombined memory block in the receiving node.
 2. The method of claim 1further comprising: calibrating a first time in the transmitting nodeand a second time in the receiving node based on a current time in anode having a calibrated shared reference clock wherein the current timefrom the node having the calibrated shared reference clock is receivedin the transmitting node and in the receiving node in response to acalibration request made by the transmitting node and by the receivingnode.
 3. The method of claim 1 further comprising: extracting andanalyzing tagged data stored within data packets at the receiving nodefor determination of communication path one way times and WAN linkcongestion.
 4. The method of claim 1 further comprising: extracting andanalyzing the tagged data stored within data packets at the receivingnode for determination of packet loss.
 5. The method of claim 1 furthercomprising: selecting multiple communication paths in response to priorpath selection.
 6. The method of claim 1 further comprising:transmitting a duplicate of the first fragment across a thirdcommunication path to the receiving node and transmitting a duplicate ofthe second fragment across a fourth communication path to the receivingnode; selecting the first fragment or the duplicate of the firstfragment as the first received first fragment in the receiving node andselecting the second fragment or the duplicate of the second fragment asthe first received second fragment in the receiving node; and combiningthe first received first fragment and the first received second fragmentas the combined compressed report.
 7. A non-transitory computer readablemedium having stored thereon executable instructions that when executedby the processor of a computer control the computer to perform stepscomprising: fragmenting a block of memory having a transmission sizeexceeding a transmission capacity of a first communication path into afirst fragment and a second fragment; transmitting the first fragmentfrom a transmitting node across the first communication path to areceiving node and transmitting the second fragment from thetransmitting node across a second communication path to the receivingnode, the second communication path selected from multiple communicationpaths as having a highest measure of independence among the multiplecommunication paths; receiving the first fragment and the secondfragment in the receiving node as a combined memory block; and storingthe combined memory block at a location in a distributed memory in thereceiving node, the location locked to assure full coherency of thecombined memory block in the receiving node.
 8. The non-transitorycomputer readable medium of claim 7 further comprising: calibrating afirst time in the transmitting node and a second time in the receivingnode based on a current time in a node having a calibrated sharedreference clock wherein the current time from the node having thecalibrated shared reference clock is received in the transmitting nodeand in the receiving node in response to a calibration request made bythe transmitting node and by the receiving node.
 9. The non-transitorycomputer readable medium of claim 7 further comprising: extracting andanalyzing tagged data stored within data packets at the receiving nodefor determination of communication path one way times and WAN linkcongestion.
 10. The non-transitory computer readable medium of claim 7further comprising: extracting and analyzing the tagged data storedwithin data packets at the receiving node for determination of packetloss.
 11. The non-transitory computer readable medium of claim 7 furthercomprising: selecting multiple communication paths in response to priorpath selection.
 12. The non-transitory computer readable medium of claim7 further comprising: transmitting a duplicate of the first fragmentacross a third communication path to the receiving node and transmittinga duplicate of the second fragment across a fourth communication path tothe receiving node; selecting the first fragment or the duplicate of thefirst fragment as the first received first fragment in the receivingnode and selecting the second fragment or the duplicate of the secondfragment as the first received second fragment in the receiving node;and combining the first received first fragment and the first receivedsecond fragment as the combined compressed report.